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DoD  Weapon  Systems  Acquisition  Still  a  High 
Risk  Area 


While  DOD’s  acquisition  process  has  produced  the  best  weapons  in  the  world,  it 
also  yields  undesirable  consequences  in  weapon  system  programs  -  cost 
increases,  schedule  delays,  and  performance  shortfalls 

Problems  occur  because  weapon  programs  do  not  capture  early  on  the  requisite 
knowledge  that  is  needed  to  effectively  manage  risks 

Programs  move  forward  with  unrealistic  cost  and  schedules  estimates,  lack 
clearly  defined  and  stable  requirements,  use  immature  technologies,  and  fail  to 
solidify  design  and  manufacturing  processes  at  appropriate  junctures  in 
development 

As  a  result,  programs  require  more  resources  than  planned,  the  buying  power  of 
the  defense  dollar  is  reduced,  and  funds  are  not  available  for  other  competing 
needs 


Paul  Francis,  Director 
Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
May  18,  2004 
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Problem  Statement 


•  A  statement  of  organizational  process  maturity  or  capability  level  does 
not  guarantee  performance  to  that  same  level  of  proficiency  on  an 
individual  project 

•  Most  DoD  contractors  claim  high  maturity/capability  levels,  yet  from  the 
perspective  of  the  acquirer,  systems  engineering  and  program 
management  practices  are  severely  lacking 

•  Teaming  arrangements  further  cloud  the  issue  of  process  execution 
and  proficiency 

•  Associated  problems  may  not  be  evident  until  significant  cost, 
schedule,  or  performance  objectives  have  been  missed  at  a  late  point 
in  the  program  where  corrective  actions  are  very  costly 

How  can  the  acquirer  gain  the  necessary  insight  into  process 
execution  and  proficiency  as  weii  as  reinforce  desired  behaviors?  / 
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The  Mission  Planning  Solution  and  What  was  Learned 


A  “simple”  solution  -  solve  the  problem  through  contract  requirements 

•  All  contractors  must  be  at  least  CMM  maturity  level  3  to  bid  on  IDIQ 
software  development  contracts 

•  All  winning  contractors  must  achieve  CMMI  maturity  level  3  NLT  24 
months  after  contract  award 

Monitor  the  contracts  to  assure  that  CMMI  requirement  is  met  by 
deadline 

Conduct  a  round  of  CMMI  baseline  SCAMPI-Bs  for  Contract  Monitoring 
to  assist  process  improvement  toward  meeting  that  goal 


The  process  of  conducting  5  baseline  SCAMPI-Bs  for  Contract 
Monitoring  was  an  “eye  opening ”  experience 
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What  is  SCAMPI-B  for 
Contract  Monitoring 
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High  Maturity  ^  High  Maturity 

Organizations  Projects 


•  Maturity  Levels  are  indicators  of 
organizational  potential 
performance 

•  They  describe  how  the  next 
project  may  perform  based  on  a 
sampling  of  existing  projects 

•  Maturity  Levels  reside  at  the 
organizational  level  and  are  not  an 
indication  of  how  an  individual 
project  is  performing 
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Standard  CMMI  Appraisal  Method  for  Process 
Improvement  (SCAMPI) 
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Benchmarking  and 
Baselining 


Tailorable  Assessment 
“Mini”  Appraisal 
as  executed 


Quick  Look 

‘Intentions”  for  execution 


No  ratings 


Maturity  or  Capability 
ratings 


No  ratings 


Cost/Duration 
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SCAMPI-B  for  Contract  Monitoring  Ground 
Rules  - 1 


Use  the  process  model  -  CMMI 

•  Interview  questions  based  on  model 

Appraisal  of  process  performance  and  adherence 

Focus  on  risk  assessment  -  risks  associated  with  process  performance, 
adherence,  and  capability 
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SCAMPI-B  for  Contract  Monitoring  Ground 
Rules  -  2 


Observe  strict  confidentiality 

•  Results  not  attributable  to  individuals  or  interview  groups 
Approach  SCAMPI  collaboratively 

Results  in  actionable  findings  by  Program  Office  and/or  Contractor 
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SCAMPI-B  for  Contract 
Monitoring  for  MPEC 
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Mission  Planning  Contracting  Structure 


IDIQ 


Delivery  Orders 
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How  SCAMPI-B  for  Contract  Monitoring  was 
Employed  for  MPEC  - 1 


Used  appraisals  to  baseline  compliance  with  CMMI  requirement 

•  Demonstrate  capability  or 

•  Develop  findings  which  yield  improvement  opportunities 

One  project  would  be  selected  from 
each  contractor  as  a  ‘representative’ 
of  all  Mission  Planning  projects  for 
that  contractor 
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How  SCAMPI-B  for  Contract  Monitoring  was 
Employed  for  MPEC  -  2 


Appraisal  findings  would  be  used  to  assess  progress  toward 
meeting  this  the  24  month  CMMI  requirement 

•  Results  to  be  factored  into  future  DO  competitions 

Appraisal  finding  to  be  used  for  contract  monitoring  to  identify 
areas  of  risk  in  project  execution 

Appraisal  findings  resulted  in  request  for  Process  Improvement 
Plans  from  the  contractors 

•  Way  ahead  to  fix  findings  in  representative  project 

•  Way  ahead  to  fix  similar  known  deficiencies  in  all  Mission 
Planning  work 


>4/1 
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Program  Office  Changes 
Resulting  from  SCAMPI-B 
for  Contract  Monitoring 
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Results  Changed  the  Way  the  Program  Office 
Does  Business  - 1 


•  Contractor  Process  Improvement  Plans  and  status  are  tied  to 
contractor  award  fee 

•  RFP  language  has  been  modified  to  better  reflect  the  program  office 
desire  for  CMMI  compliance  across  development  teams  -  including 
subcontractors 

•  Based  on  input  from  appraisals,  the 
program  office  is  reviewing/modifying 
the  standard  CDRL  list  to  get  best  ROI 

•  Business  and  technical  rules  for  MP 
developers  have  been  modified 

oriji 
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Results  Changed  the  Way  the  Program  Office 
Does  Business  -  2 


•  Program  Office  is  modifying  internal 
processes  to  better  take  advantage  of 
information  provided  through  contractor’s 
standard  processes  (e.g.  metrics) 


“In  light  of  significant  personnel  cuts,  we  need  to 
provide  a  smart  way  to  provide  a  laser-like  focus  to 
our  key  issues.  SCAMPi-B  for  Contract  Monitoring 
reviews  and  EVM  analysis  provide  that  focus  for  us.  ” 

Steven  A.  Cote,  GS-15,  DAF 
Deputy  Director,  951st  Electronic  Systems  Group 
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Lessons  Learned 
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Positive  Lessons  /  Benefits  - 1 


Overcoming  common  misperceptions  program 
offices  have  about  CMMI 

•  What  does  a  maturity  level  mean  -  how 

is  it  attained 

•  Need  to  understand  the  scope  of 
previous  appraisals  and  certifications 

—  Organizational  unit  appraised 
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Positive  Lessons  /  Benefits  -  2 


More  savvy  consumers 

•  Monitoring  is  more  necessary 
than  people  think 

•  CMMI  compliance  and  appraisal 
finding  resolution  should  be  tied 
into  program  office  business 
activities 

—  Award  Fee  Criteria 
(powerful  motivator) 


—  Business  and  Technical  Rules 


—  Proposal  Evaluation  Criteria 


■Mi 
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Positive  Lessons  /  Benefits  -  3 


Appraisal  results  can  be  used  to  assist  program  management 

•  PMRs  can  focus  on  areas  of  concern  based  on  findings 

Uncovering  program  risks  is  more  important  than  specific 
maturity  levels 


Real-time  contract  monitoring  is  key 
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Hard  Lessons  /  Ongoing  Challenges  - 1 


Spirit  vs.  letter  of  the  model 

•  Assure  intent  of  the  model  is  met 
versus  using  “checklist”  mentality 

•  Determine  consistent  approach  to 
dealing  with  corporate  “process 
lawyers” 

•  Shouldn’t  judge  “goodness,”  but  you 
should  judge  “reasonableness” 


4 
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Hard  Lessons  /  Ongoing  Challenges  -  2 


Variability  in  the  process  itself,  within  the  allowable  scope  of  the 
method,  can  itself  modify  results 


•  Teams  take  on  personalities  based  on  membership  (team  members 
and  lead  appraisers) 


Different  personalities  can  provide  inconsistent  results 


Need  to  ensure  that  decision 
criteria  are  consistently  applied 

Having  a  core  group  as  a  subset 
of  each  appraisal  team  provided  a 

consistent  methodology  and 
interpretation  of  evidence 
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Future  Directions 
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Way  Ahead  -1 


Mission  Planning  is  reviewing  options  for  verification  of  the  November 
2006  CMMI  Maturity  Level  3  requirement 

•  Accept  results  of  contractor  SCAMPI  As? 

—  Review  of  detailed  appraisal  results 

•  Conduct  Program  Office  sponsored 
SCAMPI  As? 

—  Requires  core  team  to  re-appraise 
contractors 

•  Verify  completion  of  all  process 
improvement  activities  outlined  in 
approved  Process  Improvement  Plan 
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Way  Ahead  -  2 


How  do  we  “maintain”  the  process  improvement/process  maturity  focus 
to  ensure  consistent  levels  of  performance/execution? 

•  The  Program  Office  is  working  to  develop  a  multi-year  plan  for  continual 
assessment/monitoring 

—  Additional  appraisal  for  all  contractors  (cost/benefit) 

—  “Spot  check”  areas  of  known  deficiency 

—  Appraisals  for  projects  with  performance  issues 

—  Appraisals  for  projects  that  fall  within  another  organizational  unit 
than  “representative”  project 


Ay, 
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Way  Ahead  -  3 


Beginning  to  analyze  SCAMPI-B  for  Contract  Monitoring  results  against 
product  quality  to  determine  if  there  are  correlations 

•  Could  impact  risk  management/corrective  actions  in  the  future 
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Summary 


A? 

□m 


SCAMPI-B  for  Contract  Monitoring 

-  Software  Engineering  Institute 

Carnegie  Mellon 

Lorraine  Adams  and  Kathy  Bastien 
March  2007 

28 

©  2006  Carnegie  Mellon  University 

Summary  - 1 


SCAMPI-Bs  for  Contract  Monitoring  were 
very  positive  for  Mission  Planning 

•  Provided  insight  into  contractor 
processes  and  potential  program  risks 

Consistency  is  key 

Appraisal  results  have  become  a  useful 
source  of  data  to  assist  in  program 
management  activities 

•  PMs  can  focus  attention  on  areas  of  the 
project  that  have  highest  risk  due  to 
inconsistent  (or  non-existent)  standard 
processes 
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Summary  -  2 


SCAMPI-Bs  for  Contract  Monitoring  started  as  a  way  to  do  a  “quick 
check”  of  contractor  CMMI  requirement  compliance  -  and  have  now 
become  a  powerful  management  tool 

Savvy  consumers  can  utilize  SCAMPI-Bs  for  Contract 
Monitoring  as  part  of  their  management  “tool  box ” 
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Contact  Information 


Lorraine  Adams 

Member  of  the  Technical  Staff 
Software  Engineering  Institute 
4500  5th  Avenue 
Pittsburgh,  PA  15213 
412-268-7777 
ladams@sei.cmu.edu 
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